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ABSTRACT 

Faculty members 1 reactions to constructing and 
evaluating course portfolios were examined, and how that process 
might be improved was explored as part of an annual evaluation of 
teaching. Three college departments (sociology, accounting, and 
dental hygiene and assisting) volunteered to use portfolio review in 
the 1991-92 school year. In all, 22 faculty members from these 
departments were interviewed about the portfolio development process 
(approximately 53 to 64 percent of the faculty members in each 
department). Overall, the portfolio development process did not work 
well. Most faculty members felt that they did not get anything out of 
constructing the course portfolios, that the portfolios did not 
capture their ideas about teaching, and that making judgments about 
the portfolios for evaluation purposes was difficult. Faculty members 
generally thought that the criteria were not explicit enough to use 
for faculty evaluation. Anytime a department chooses to conduct a 
portfolio review, it must recognize the time-consuming nature of the 
portfolio construction and evaluation process. Some suggestions are 
made for implementing course portfolios without using them for 
accountability purposes too early in the process. (SLD) 



**************************************^^^ 

* Reproductions supplied by EDRS are the best that can be made * 

from the original document. * 

************************************* 



Presented at the annual meeting of the American Educational 
Research Association (AERA1 . Atlanta. GA. April 13 . 1993 . 



Faculty Orientations Toward Teaching and the Use 
of Teaching Portfolios for Evaluating and Improving 
us. department of coucation University-Level Instruction -permission to reproduce this 

Ott.ce <x educational Reeearcn and improvement MATERIAL HAS BEEN GRANTED BY 

EDUCATIONAL RESOURCES INFORMATION _ ^ A 

J CENTER ,EHK3 by ftoS/^g/Q 

GrThis document net been reproduced as — t - 

received from the peraon or organization 

originating it -r ^ PriVl "1 T"l COTl 

O Minor changes nave baen made to improve U a^A. I^UWlIlbUIl . 

reproduction quality Dominion University 

a points of view or opinions stated m mis docu- jq THE EDUCATIONAL RESOURCES 

ment do not necessarily repreaant oHiCiai iu e A D ii*Tinu ocmtcd /cam « 

OERi position or policy INFORMATION CENTER (ERIC). 

Much recent attention has been directed toward the use of 
teaching portfolios as a means of evaluating and improving 
university-level instruction. Seldin (1993a) notes that in the 
past four years the number of institutions using some form of 
portfolio review has risen from ten to over 400 and that it is used 
in settings as diverse as the University of Maryland, Ball State 
University, Texas A&M University, Miami-Dade Community College, 
Columbia College in South Carolina, Harvard University,^ and 
Dalhousie University in Canada. The American Association of Higher 
Education (AAHE) as part of 11 The Teaching Initiative" project 
promotes the use of teaching portfolio as a means of capturing the 
complexities of teaching, of placing the responsibility for 
evaluating teaching in the hands of faculty, of prompting more 
reflective practice and improvement, and of fostering a culture of 
teaching and new discourse about it (Edgerton, Hutchings & Quinlan, 
1991) . Teaching portfolios are advocated by Seldin (1991) as a 
means of documenting the quality of teaching performance while 
preserving the complexity and individuality of teaching and of 
providing evidence equivalent to that provided in evaluating the 
quality of research for tenure and promotion decisions. While not 
without flaws, he sees portfolio review as the best alternative we 
currently have for representing a faculty members quality of 
teaching. 

The impetus for the use of teaching portfolios has come from 
several sources. The need to improve undergraduate education in 
response to changes in higher education (Green, 1990) and the demand 
for greater faculty accountability (Seldin, 1990) have prompted the 
need to find a better way to improve teaching and at the same time 
document or make a case for effective teaching. Changes in 
perspectives on what constitutes scholarship (Boyer, 1990) and the 
manner in which effective teaching evolves through reflective 
processes (Shulman, 1989) have motivated the use of teaching 
portfolios as a context specific means of displaying the 
transformational processes and products of the evolving craft of 
teaching. 



The present study sought to examine faculty members 1 reactions 



to constructing and evaluating course portfolios and how that 
process might be improved as part of an annual evaluation of 
teaching. As faculty members from three departments in the 
university were interviewed about their initial experiences with 
portfolio construction and evaluation, serious questions arouse as 
to whether portfolio review will work as a system of accountability 
or improvement. They questioned whether portfolio construction and 
review could capture their ideas about effective teaching or be of 
sufficient benefit for improving their teaching to justify the 
amount of time required for constructing and reviewing them. 

BACKGROUND TO THE PRESENT STUDY 

My interest in the use of teaching portfolios started in 1989 
as a result of participating in a faculty senate committee that 
sought to improve the university 1 s existing system of teacher 
evaluation. The committee reviewed several articles and books 
focusing on the evaluation and improvement of teaching and 
recommended that peer review of course portfolios be conducted 
annually as a means of evaluation supplementary to that provided by 
student ratings. Our recommendation was based on the idea that 
while student ratings could be justified as valid and reliable 
measures of student instructional needs, course portfolios would 
better reflect the appropriateness of the intellectual demands of 
the course and what faculty did to help students meet those 
demands • 

Following approval by the faculty senate and central 
administration, a second committee used suggestions from Witman 
and Weiss (1982) as guidelines for developing the course portfolio 
review process. Course portfolios were to be evaluated annually in 
each department and were to contain information on teaching 
responsibilities, grade distributions given in courses, course 
syllabi, handouts, assignments, materials used, activities, and 
exams. Also, portfolios were to contain a discussion of the means 
by which faculty helped students meet the intellectual demands of 
the course and evidence to support those claims. The entire effort 
was initiated, sponsored, reviewed and approved by the faculty 
senate with support from central administration. 

THE PRESENT STUDY 

Three departments volunteered to use portfolio review during 
the fall and spring semesters of the 1991-1992 academic year. I 
initially contacted the department chairs from each department to 
get approval to conduct interviews with their faculty. Subsequent 
to that approval I selected a systematic sample of faculty members 
in the department (every nth faculty member) from the university 
phone directory. However, some of those faculty selected were on 
leave, others had left, and other key faculty members emerged from 
the interviewing process. Key faculty members included individuals 
who had chaired or participated on committees examining the peer 
review process or were untenured. In all, 22 faculty were 
interviewed. In selecting these faculty I consciously attempted to 
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insure that they were representative of different groups, tenured 
verses nontenured, male verses female, and held different ranks in 
the department. I then contacted each faculty member to set up 
what I thought would be a 45 minute to one hour interview. In 
actuality, many of the interviews lasted from 45 minutes to two 
hours. In setting up the interviews I indicated that my purpose 
was three fold: to do personal research, to collect information 
that might be useful for a pending pre-proposal , and to find out 
how they were working for possible recommendations for modifying 
the portfolio review process. Of the 23 faculty members contacted 
only one refused to participate. In all, 53% to 64% of the faculty 
members in each department were interviewed. 

At the beginning of each interview I again explained the 
purpose of the interview, assured each faculty member of anonymity, 
and requested that the interview be tape recorded. Three faculty 
members declined to have the interviews recorded but were 
comfortable with notes being taken and offered to clarify any 
questions later. All tapes were replayed and field notes reread. 
Tapped and written interviews were then either fully transcribed or 
significant portions transcribed by myself. Faculty members' 
concerns and perceptions were very apparent from the interview 
process. However, making sense of them and interpreting what went 
wrong came from reviewing recent literature and attending the 
recent national symposium held in San Antonio Texas on portfolio 
review. 

Questions explored in the interviews 

A central focus of the interviews was to find out how 
important teaching was to faculty, what faculty members did that 
they considered good or effective teaching, whether they felt that 
came out in the portfolio review process, and if they felt they 
could or should evaluate peers' portfolios. The following 
questions were used in the interview process. The questions used 
were modified from an initial set of questions as interviews were 
conducted in response to faculty members concerns about 
accountability issues and the utility of the portfolio review 
process. The wording of each question is approximate as questions 
were slightly modified, further explained or expanded on during the 
interview process. 

la. How did you become a professor? 

lb. Personally, to what degree is teaching important to you? 
lc. Professionally, to what degree do you think teaching is 
important? 

2a. What do you think about when you think about good 
teaching? 

2b. How do you think that is best accomplished? 
2c. What do you do to accomplish that? 
3a. What did you include in your portfolios and why? 
3b. How much time was spent preparing it? 
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3c. How much time would it take you to revise your portfolio 
for future use? 

4a. Did portfolio review cause you to reflect on what you do 

to be an effective teacher? 
4b. Did it generate or call to mind questions to explore? 
4c. Did you change anything as a result of constructing 

your portfolio? 

4d. Do you feel that portfolio review captured what you think 
about when you think about good teaching? 

5a. Did peers or a committee review your portfolio? What 
happened? 

5b. What was emphasized? 

5c. What was your reaction to peer or committee comments? 
5d. How did this tie into your personal definition of good 
teaching? 

6a. What is your departments orientation toward teaching? 
7a. How should portfolios be evaluated? 
7b. Could you evaluate a peer f s portfolio? 

7c. How comfortable would you feel evaluating peer's 
portfolios? 

8a. Can portfolios portray an overall picture of teaching 

effectiveness? 
8b. Can preparing portfolios help improve teaching? 
9a. Was portfolio construction and/or evaluation practical? 
9b. Was portfolio construction and/or evaluation worthwhile? 

Some methodological concerns 

Before presenting the findings some methodological concerns 
need to be addressed. One is the relationship of the author to 
faculty being interviewed and whether the dialogue that occurred 
was candid. As I am not in any kind of administrative position and 
mainly associated with the faculty senate as a colleague, the 
interviews should not have been threatening to the faculty member. 
However, I am associated with advancing the use of portfolios as a 
means of evaluation, and therefore concerned that faculty might be 
reserved in their responses. Frankly, as faculty discussed the 
idea of using portfolio review I was apologized to a lot, usually 
in the context of "sorry but I don f t think this is a good idea". 
They also indicated that what was said on tape had to be 
confidential otherwise they did not see any point to the 
interviews. As representativeness of the faculty participating was 
not a major concern as I was more interested in gaining insight 
into their reactions to the portfolio review process, nevertheless, 
the faculty participating appeared to be reasonably representative 
of faculty in these departments. 

Another contextual concern is the general level of teaching in 
these departments. All three departments value and view themselves 
as good teaching departments. In the case of two of these 
departments there is objective information to support effective 
teaching claims. The department (school) of Dental Hygiene and 



5 



5 



Dental Assisting has for some years received the highest scores on 
the national boards for a department of its type not associated 
with a dental school. The Accounting department has passing rates 
on the CPA exam equivalent to those of the two top academic schools 
in the state despite the fact that the university accepts a fairly 
average student. A student from that program received exceptional 
recognition this past year for obtaining the highest score in the 
country on the CPA exam. The sociology department is generally 
recognized in its college and on campus as a very good teaching 
department. 

FINDINGS 

Reasons for becoming a professor 

Faculty members varied greatly in terms of how they came to be 
professors. Some simply "stumbled into it", others "liked the life 
style", and still others "had aspired to be teachers since they 
were 7 years old". A common thread that ran through most of these 
conversations was that once these individuals had been in a 
situation where they had taught they became "hooked on teaching". 
Several of the faculty members interviewed indicated that they had 
given up and continue to give up much higher paying jobs in 
business and the public sector because they liked teaching and made 
enough money to be comfortable. 

Personal importance of teaching 

Faculty personally viewed teaching as "very important" to 
"extremely important" or in some cases said "it's my life". In 
general most faculty verbally ascribed a great deal of personal 
importance to teaching. A number of faculty members indicated that 
they liked doing research and some saw teaching and research as 
commingled processes. However, they all ranked teaching as most 
important personally compared to research and service. 

Professional importance of teaching 

When asked if they felt teaching was important for 
professional advancement and rewards most said that 11 it wasn't". 
Research and acquiring grants were viewed as the ma^or means of 
getting rewards, tenure and promotion. However, they did see 
teaching as an very important priority in their departments and 
expressed the idea that good, not fair, teaching was expected. 
Anything less than that would be unacceptable for tenure and 
promotion and would affect salary decisions negatively. With the 
exception of a few faculty members, they did not view effective or 
good teaching as being rewarded professionally. Only if teaching 
was substandard could it have an effect, and then only a negative 
one. Faculty members of all three departments viewed their 
departments as placing a high priority on research and grants. 
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Definitions of good teaching 

When I initially started this research I expected to find at 
least some faculty stressing acquiring content knowledge as their 
primary goal in teaching. What I found was concern by faculty 
members interviewed for developing students 1 understanding , 
perspectives, attitudes, critical thinking skills, problem solving 
skills, and problem approach behaviors. Also, a number of faculty 
members indicated that helping students in their personal and 
professional growth was an important goal of theirs. Faculty 
members thought that these goals were best accomplished in a 
variety of ways. In fact, there were 22 different styles of 
teaching which could not be easily characterized. Some styles of 
teaching revolved around a few orientations toward teaching, others 
were very eclectic. What follows is a description of parts of some 
of those. Pseudonyms were chosen as a way of characterizing the 
different styles of teaching I observed. 

The applied realist was someone who continually used concrete 
and humorous stories, metaphors and analogies to make the concepts 
he was teaching more accessible to students. He did this in the 
kinds of ways students would be expected to use it in analogous 
real life situations while associating positive affect with the 
cognitive learning. I should mention here that he never used 
technical educational jargon to explain what he was doing and that 
I had to keep questioning him in order to formulate my ideas about 
what he was doing. 

The applied realist felt good teaching was a matter of 
"entertaining as well as educating" and keeping students "focused 
on what it is you 1 re doing" and thought that was best accomplished 
by using "humor" . When I explored what he meant by that I found 
that he continually captured general principles of what he was 
teaching in humorous stories. For example he related how "good 
businesses to control and protect their assets have what's called 
a system of internal control" and how going to the movie theater 
where your ticket is torn up so that it can't be resold 
demonstrates that principle. ■ He then went on to relate how he told 
a story to his students about totally embarrassing his wife at a 
high school football game by insisting that the ticket taker tear 
up the ticket while several people waited in line. His point to 
his students was "they should be seeing and noticing these kinds of 
things in their everyday lives". The humorous stories he told were 
used to cue students to remember those principles. He constantly 
related textbook principles to practical real life situations. He 
was always looking for new ways to get information across using 
concrete real life situations. Another example he used was how it 
was difficult for first year students to grasp the concept of 
estimating dollar values of unfinished products in continuous 
process manufacturing. He related that he had read an article 
where a professor had students use tinker toys in an assembly line 
process in class and then had them stop and go through the process 
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of estimating dollar values of unfinished products. He hadn't 
tried it yet but then pointed to a hugh box of tinker toys setting 
on his file cabinet. 

The reflective practitioner constantly thought through what 
she did in her courses, why, how it worked, and reviewed and 
revised what she did in the context of educational and adult 
learning theory. She could easily articulate what she did and why. 



The reflective practitioner when discussing effective teaching 
said "I work very, very hard to make class interesting; I never 
just lecture, I use lots of different media; we like to discuss and 
use lots of interaction and activity, what we call adult learning 
strategies, I try very hard to incorporate those into my classes". 
When asked how she thought that was best accomplished she replied 
"I think it all starts with preparation. I don't think there's many 
good teachers who just get up and talk off the cuff. I think it 
takes preparation, especially if your going to have good 
interaction. If you want your students to do group activities you 
have to plan in advance for this stuff, you need objectives-how are 
they going to accomplish these objectives, unless your model of 
good teaching is just spitting out information all the time, which 
I don't think is good teaching". I then asked, so preparation is 
one thing? She responded, "preparation, variety, being open to 
students ideas and students' viewpoints, not taking the role of I 
am the boss, its going to be my way no matter what. If I am in 
front of the class it needs to be an opportunity for exchange, for 
the students and the professor. I think that makes good teaching, 
you have to be willing to do that in the classroom-relinquish 
control". Later when discussing preparing portfolios she went on 
to say "I think about what I am doing and why every time before I 
do my lecture, which is why I am always updating my course 
portfolio. Having to write down that I am doing it or I am not 
doing it, to me personally is a waste of time-its redundant-for 
people who don't update or write down what they do it might have 
more value". 

The perspective taker sought to challenge, expand and enrich 
the ways students view things. Also, she focused on developing the 
students' skills in assessing and processing information. 

The perspective taker when asked about what she considered 
good teaching said: "I am probably a little different in how I 
define good teaching. To me, to be a good teacher you have to be 
a good researcher. The person who has never done research but just 
repeats what other people say may be entertaining, but won't 
necessarily have the incentive to keep up. They may not have the 
knowledge, the most recent cutting edge knowledge, won't have the 
experiences. I can talk about culture shock because I go through 
it every summer when I go to .... I can talk about things because 
I do them, I can talk about genealogies because I do them. One of 
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the things to be a truly good teacher, not necessarily an 
entertaining one, but a good one where students actually learn the 
material and really understand the discipline, you have to be a 
good researcher". When I asked how she accomplished this she said: 
"I show slides from my field work, I show examples of things in the 
book and relate them to my own research and how I experienced it 
and I give examples of what it's like if you have more than one 
wife, like in polygamous societies. I talk about how people 
interact and deal with things like co-wife jealousies and chores 
and competition. So I try to use examples from my own experience. 
When I try to get them to see what it's like to have a different 
culture or religion I show them slides of a ritual. I show lots 
of slides of my field work, talk about it from the perspective of 
the Indians not from an outsider or •••(academic discipline). We 
discuss cultural relativity, so that students can get a feeling 
that this is meaningful or right or correct to natives just as much 
as their Christianity may be to the students. I think research is 
absolutely critical to being a good teacher". She also mentioned 
developing critical thinking skills and when I asked her how she 
did this she said: "I give them different perspectives and I 
develop each perspective — conflicting ones and then I ask them how 
do we decide which one has the most data to support it? Which one 
is valid? So I make them go through the thinking process of 
choosing. What are the problems of this one, of that one? How do 
we settle the issue?" She went on to relate "In my upper division 
courses I have them read books that are totally opposed, then I 
have them write a paper on why they are choosing different types of 
data like in one book, and other types of data in another book, to 
discuss the same point. I make sure the books are discussing the 
same issue but coming from different theoretical perspectives so 
they see the influence of one's theoretical orientation on what one 
considers valid and what one considers evidence". 

I found this attention to developing perspectives through 
demonstrations and discussions in a number of interviews. All the 
faculty members I interviewed were concerned with teaching students 
ways of approaching and analyzing information. They might 
accomplish this through dialoguing, debating, written or computer 
exercises, case studies, games, or intellectual modelling. None of 
the faculty members interviewed saw effective teaching as just a 
matter of conveying information. 

I did not find any easy categorical way of characterizing the 
problem solver . The best way to describe it was that he used a mix 
of ways to approach the problems that arise in teaching* These 
approaches did not seem to come from three or four central tenants 
about teaching but rather from a very eclectic and pragmatic 
orientation. He focused on developing understanding and tried to 
relate what he did in the classroom to real life situations 
students would face. Much of the information on what he did came 
out after prolonged questioning about his ideas about good 
teaching. 
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The problem solver indicated that he was currently teaching a 
televised course and spending about 35 hours a week teaching and 
developing that course. He saw learning to teach on interactive 
television as important because he saw that as a new focus of the 
university and he thought a couple of articles might come out of 
it. He went on to say "there are a lot of differences (teaching on 
TV) , you really have two audiences. You have the students and then 
you have the channel flippers because it's going out over ...(a 
cable network) and not that you worry about them except for the 
fact that they will be evaluating ODU based on what they're looking 
at, and they 1 re not sitting there taking notes like the students 
are, and when you talk slow and they've just been listing to CBS 
news or something else and it looks like-oh boy that's way they 
teach at ODU. So I've been experimenting with different things in 
the course that doesn't come across on cable as poorly as it might 
come across in the classroom even though you're doing an excellent 
job with the students". When I asked about these things he went on 
to relate how, when students were taking notes, that instead of 
slowing down as he would usually do, he would repeat himself or use 
"fill in" information that students didn't need to take notes on. 
He also noted that students could tape these presentations and take 
them home and replay them, especially if they encountered 
difficultly understanding some of the ideas presented. He was 
currently struggling with the problem of interaction and students 
asking questions because of the intimidation of appearing on 
interactive TV. His goals in teaching were to take students to a 
much higher level of understanding, have them enjoy the course, 
teach a rigorous course and have students learn a little more about 
the environment that they would be working in. When asked how he 
did this he remarked that the "basics" were easily gotten from the 
textbook but that these ideas needed to be related to real life 
work experiences and what actually happens. He might use "cases" 
or work situations to develop ideas or explain to students how 
comments they could expect on their evaluations in work related 
situations should be interpreted. He was currently putting 
together an advanced seminar course where students would argue a 
point with another student as in a court of law. Both points of 
view could be supported equally and the judge would be a member of 
a local business firm with whom they might seek employment later. 

What happened when portfolios were constructed and reviewed? 

Basically, it did not work. For the majority of faculty 
members, across all three departments, when I asked faculty if they 
got anything out of constructing course portfolios for the 
evaluation process they said "no". When I asked whether the 
portfolios had captured their ideas about teaching they said "it 
didn f t". When I asked what questions or ideas it had generated~"it 
hadn't". When I asked if they could judge differences in quality, 
most said they "couldn't". They viewed most differences that they 
had observed as a matter of "style". While some faculty members 
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felt they could judge differences in quality, almost all faculty 
felt uncomfortable or very uncomfortable making those judgements 
for evaluative purposes. Constructing portfolios did cause faculty 
members to think about the content and requirements in their 
courses against departmental standards, but these concerns were not 
tied to their personal definitions of effective teaching. Only a 
few faculty members reported any benefit from putting course 
portfolios together in this sense, and these tended to be faculty 
members who had written some reflective thoughts about what their 
objectives were, what they did to accomplish those and how that was 
reflected in their course portfolios. However, even those 
individuals did not feel that it captured their ideas about 
effective teaching. What happened in each of the departments might 
be informative . 

In the accounting department faculty portfolios were reviewed 
by an undergraduate and graduate committee depending on the level 
of the course. Faculty were to submit their course syllabi, 
handouts, materials used in the course, course grade distributions, 
assignments and exams. They did not submit reflections on their 
philosophies of teaching, objectives or any discussion of what they 
did in their courses to accomplish those objectives. The major use 
of the portfolio review process was to insure that each course 
covered what it was supposed to according to the departmental 
syllabi for the course and meet certain departmental concerns. 
These departmental concerns focused on th^ appropriate use of 
computers and the level and types of assignments and exams used. 
'Committees checked courses against departmental syllabi and 
examined portfolios for these departmental concerns. They also 
checked very closely whether what was on exams coincided with what 
was taught in the course. Evaluative comments then tended to 
address these issues, but only if the faculty member was deficient 
in some area. As a couple of faculty members commented "you have 
to understand that accountants are used to looking for what f s 
wrong. If everything is okay then they don't say anything". Most 
faculty were found to be teaching up to the departmental standards 
but deficient in some areas. Faculty did not find these comments 
helpful or relevant to their personal definitions of effective 
teaching. In most cases the criticisms resulted more from the 
faculty member failing to adequately explain what they were doing 
in their course portfolios. 

In the dental hygiene department all faculty members keep 
portfolios and have been doing so for more than five years. Their 
portfolios contain everything they use in the class: syllabi, 
objectives, handouts, materials used, assignments, tests, lecture 
notes, overheads, etc. The typical portfolio is from six inches to 
a foot thick. They are not expected to be typed or kept in a neat 
or formal manner. In fact, that is discouraged. They are kept for 
three reasons. One is so that if someone else new to teaching the 
course needs to get started they have something to work from. The 
second reason, reported by all the faculty interviewed in that 
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department , was "for myself 11 . Faculty members found portfolios to 
be a good base from which to work* They continually used them to 
organize themselves, provide a basis for thinking about their 
teaching and a depository for ideas and assignments. The third 
reason is so that they can be used by the department chair for 
evaluating teaching. The portfolios did not contain a statement of 
teaching philosophy or any reflective statements relating 
ob j ect ives to what is done in the class , or relate these to 
evidence of student learning. As some faculty members commented 
"it's all there you can see for yourself" and saw writing that out 
as "a waste of time" or "busy work". When faculty members were 
asked whether they got anything from the portfolio review process 
they said they did not, outside of maybe re-examining whether they 
had varied courses enough. They did that kind of thinking and 
re iewing throughout the year as they taught their courses and 
interacted with students. Using portfolios as part of an evaluation 
process did not stimulate much additional reflective thinking about 
their teaching, and did not capture the rich differences in their 
individual styles of teaching. 

The sociology department included course syllabi, materials 
and assignments, readings, objectives, exams and reflective 
statements relating their objectives to what they did in the course 
and what they asked of students on exams and assignments. A 
committee was formed before the process was started to examine 
guidelines for portfolio preparation and evaluation. After the 
department had gone through the portfolio review process, another 
committee evaluated how to revise the process. Course portfolios 
were reviewed by a committee of peers with all faculty members in 
the department serving as a member of a three to five person 
committee for each portfolio reviewed. As this was a trial run, 
faculty members agreed to only give comments back to the individual 
faculty member and there was a tacit agreement not to be overly 
critical. Some faculty members found the reflective part useful 
and reported that it helped them clarify "vague" ideas they had 
about teaching and reflect on what they were doing. They also 
reported getting some good ideas from reviewing other peoples 
portfolios. In some cases they said it helped them see the level 
of expectation in other courses and some reported that it was 
"inspiring" to see what other faculty members were doing, and that 
they had tried some of these ideas in their courses. 

However, in general there was a very negative reaction to the 
idea of portfolio review in this department, especially for 
evaluative (accountability) purposes. Faculty did not see it as 
very useful, they had a great deal of difficulty determining what 
to say in their portfolios, and continually expressed great concern 
over what criteria could or would be used to evaluate them. They 
felt that if you were going to evaluate faculty members, then you 
needed to have explicit criteria for that evaluation. If you have 
explicit criteria faculty members "will simply write to these 
criteria" and you will "ritualize" the evaluation process. If 
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faculty take these definitions seriously, you will ultimately 
"trivialize 11 teaching. Also, they saw specifying what was to be 
considered effective teaching, and what and how things should be 
taught, as an encroachment on academic freedom. They did not feel 
they got much value out of constructing portfolios, especially for 
the time invested, and that it did not reflect their ideas about 
good teaching. As one faculty member stated " the substantive, 
creative engagement in teaching is hard to get from portfolios". 
Comments from peers to be of use needed to be constructive and 
specific and most faculty members found comments made by peers to 
be too general to be of use. In addition, several faculty members 
felt that peers being critical of each other would undermine morale 
in the department. Using portfolio review for improvement purposes 
with comments shared only with the individual faculty member could 
give faculty members some good ideas about teaching, however; "you 
could get at those things through other mechanisms". Mor>t faculty 
did not feel they could judge other faculty members portfolios, and 
saw most differences in teaching as a matter of "style". Whether 
they felt they could judge differences in quality or not, almost 
all faculty members felt uncomfortable judging other faculty for 
evaluative purposes. 

When I explored alternative ways of conducting portfolio 
review for improvement or accountability, these later sentiments 
were expressed by faculty in all three departments. Faculty would 
welcome the opportunity to review each others teaching and felt 
comfortable making carefully worded constructive "suggestions" for 
improvement, but they felt the responsibility for being evaluative 
was the chairperson* s. They felt that many of the differences they 
had observed or would observe in the ways faculty members taught 
was a matter of "style" and that qualitative differences were 
difficult to assess. Also, they were not sure what they would get 
out of reviewing portfolios from content areas other than their 
own. Aside from the obvious concern for differences in concepts 
being taught, they were unsure that alternative methods they had 
observed would work in their discipline. This concern was 
expressed both within and across academic disciplines. 

Untenured faculty members were very uncomfortable with being 
critical of senior faculty for accountability or improvement 
purposes, but were very interested in seeing what senior faculty 
did in their teaching. Also, they were uncomfortable with having 
their portfolios reviewed for accountability or improvement out of 
fear of criticism or looking bad. 

Faculty members varied on how much they talked about teaching 
with each other. More than half indicated that they did not talk 
about teaching with each other. Of those that did, the focus of 
their conversations was on methodological problems of how to 
improve writing assignments, deal with televised courses etc., and 
few had ever discussed their basic ideas about teaching with a 
colleague. Discussions about teaching occurred in informal 



13 



13 



settings. Formal discussions about teaching within departments 
focused on curriculum issues. 

Was portfolio review Practical? 

Faculty members' typical reaction to this question was that it 
was not. They saw it as taking too much time for the benefit they 
felt they had derived from going through the process. Typically, 
faculty members spent two to four hours preparing their portfolios 
for each course, and some spent up to a day. Most of this time was 
spent in gathering materials or pulling them from computer files. 
The time spent in preparing reflective statements and relating 
their teaching objectives to teaching practices was not 
ascertained, but it seemed not to be a major part of the task for 
most faculty. When asked how much time it would take to update 
their portfolios for future evaluations they typically reported 
that it would take about an hour. The greatest amount of time 
expenditure came in reviewing other faculty members portfolios. 
Most faculty members did not do an extensive analysis of each 
others' portfolios. Chairs saw analyzing each faculty members 
portfolios on an yearly basis as an extremely time consuming task. 
Faculty members and chairs frequently expressed the idea that in 
preparing portfolios faculty could misrepresent what they did, or 
simply write it out according a certain guideline or some statement 
about good teaching, and not actually be teaching that way. They 
thought preparing portfolios might even take time away from 
teaching and be counterproductive. Instances were cited where 
faculty had good portfolios but were known to do a poor job in the 
classroom and vise versa. 

Was portfolio review worthwhile? 

Most faculty members were undecided on whether the process of 
portfolio review was worthwhile. Doing the reflective part of the 
portfolio was indicated as of most benefit. Faculty who had served 
on promotion and tenure committees frequently stated that they 
thought having junior faculty include reflective statements about 
their teaching would be very useful and help "contextual ize" their 
teaching for promotion and tenure decisions. Almost all faculty 
members felt that it did provide information that was different 
from that provided in student ratings and that having more sources 
of information about teaching provided for a better evaluation. 
They did see a lot of value in reviewing peers' portfolios but 
questioned how long they would gain new information after they had 
gone through that process a couple of times. Going through the 
portfolio review process had not captured their ideas about good 
teaching, and they found it most deficient in capturing the 
dynamic, creative, interactive aspects of teaching. This Was in 
reference to both planning what to teach and the process of 
instructional delivery and interaction with students. Also, they 
questioned how much benefit it would have for gaining good ideas 
for the kinds of concepts and skills they taught in their 
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particular academic disciplines. 

ANALYSIS OF MAJOR CONCERNS 

Our first mistake was to proceed too fast and to have placed 
the use of portfolio evaluation in an accountability context. 
Despite the fact that this was a faculty senate reviewed and 
recommended evaluation process, faculty found themselves in a 
situation of having to do something that could potentially affect 
them greatly and felt that it had been unfairly imposed on them. 
They had not had the opportunity to gain "ownership" of the 
process. They felt threatened and lacked clear information on how 
best to proceed. While Seldin (1993b) recommends that faculty 
become involved in the process and learn to resolve the issues 
associated with portfolio review, they had insufficient guidance to 
do so readily. There was a failure to recognize the flexible and 
individualized nature of portfolio review, both in preparing 
portfolios and using them for evaluation. The immediate concern 
was for specifiable criteria, coupled with a perception that doing 
so would misrepresent what teaching was about. Another concern 
expressed by faculty was that portfolios could not capture 
effective teaching as they defined it, especially the dynamic and 
interactive aspects of it. Best practices (Wolf, 1991) and 
innovations were not the focal points for developing portfolios and 
there was a concern for being comprehensive rather than selective 
in what went into their portfolios. Little concrete advise or 
guidance was available, and the overall holistic notion of "a union 
of insufficiencies" (Shulman, 1988) was not a perspective used in 
constructing or evaluating portfolios. Placing portfolio review in 
the context of accountability evaluation had forced faculty members 
to focus on meeting prescribed standards or standards they 
perceived as vague or inappropriate rather than on how they might 
use the process to improve their teaching. 

The use of a mentoring or other collaborative relationships in 
preparing portfolios would have been extremely helpful. Mentors 
could have been used to foster dialogue about teaching goals and 
ideas and helped the faculty member put these in writing (Annis, 
1993). Seldin (1993b) suggests using trained mentors and/or 
mentors from other departments for providing the structure to 
clarify instructional intents, identify how those are manifested in 
the teaching of the course and for matching that to evidence of its 
effect on student learning in a non-threatening context. The use 
of mentors from other departments would seem especially appropriate 
for untenured faculty. Experienced mentors from other departments 
could help untenured faculty make their best case for effective 
teaching while sharing ideas for improvement. 

Another concern voiced by faculty is that portfolios do not 
capture their individual approaches to teaching, especially the 
interactive aspects of teaching. While mentoring could do much to 
alleviate that shortcoming, it still seems to fall short. Video 
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tapping and classroom visitation by peers has been offered as a way 
of doing that, and could certainly be used, but this would call for 
fairly sophisticated and time consuming interaction between the 
faculty member and a knowledgeable mentor. 

Capturing the creative, dynamic aspects of teaching not only 
in delivering instruction but in planning and reflecting on 
teaching is difficult. Much of the means of displaying this would 
be through faculty self report focusing on how much time and effort 
they have expended on reworking and improving their courses. The 
most impressive thing I found in interviewing these faculty, that 
I had not anticipated, was the time and effort they expended in 
reading background material, deciding what to teach, critically 
analyzing how to teach that material, and critically analyzing how 
well it worked. Most of that analysis occurred informally in 
periods of self reflection and while interacting with students. I 
tried to capture that process in the characterizations in the early 
part of the findings of this paper. The very heart of effective 
teaching emanates from this ever evolving, reflective, 
transformational process Shulman (1989) calls "pedagogical content 
knowledge". Other than documentary self report evidence, how are 
faculty going to be able to relate this to enhanced student 
learning? 

Edgerton f s, Hutchings 1 and Quinlan f s (1991) approach to the 
problem of capturing the transformational and interactive aspects 
of teaching, building on the work of Shulman and his associates, is 
through presenting "episodes" of reflective teaching. Teachers 
present a context for a particular area of concern, reflect on what 
they have attempted to do about the problem, and relate it to 
student work. These "contextual ized" episodes of teaching try to 
capture how the content knowledge of the teacher is transformed by 
the teacher in interaction with a particular group of students in 
a particular teaching situation. Such episodes of teaching could be 
accumulated over time to paint a picture of effective teaching. 
Using such a process and procedure to document effective teaching 
would seem to more closely approximate what faculty members 
actually do in their teaching than any other procedure currently 
available. 

To implement such a process faculty members in departments 
could interact with a trained mentor to develop these episodes. 
However, this would require a sophisticated level of expertise on 
the part of mentors. Translating the transformation processes of 
teaching is difficult. Success depends on the motivation for 
faculty members to participate and mentors to engage in such 
interactive processes. The process itself of reflecting on 
teaching stimulated by preparing portfolios seems to have enough 
intrinsic value to motivate many faculty members to participate and 
to continue to engage in the portfolio review process (Seldin, 
1993a). Faculty members 1 accounts of why they became teachers 
("hooked on teaching") supports the notion that the intrinsic value 
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of teaching is very motivating. If the portfolio review process 
can be usefully tied to their personal definitions of effective 
teaching then sufficient motivation may result from constructing 
portfolios to engage in the process. 

The role of faculty members in the same department is another 
concern. Shulman (1992) notes that only attending to generic 
pedagogical principles fails to capitalize on what communities of 
teachers learn or could learn from each other. A possible solution 
to the problem is to have colleagues in the same academic 
disciplines act as mentors for each other. This would stimulate 
focused dialogue about teaching and capitalize on the interest of 
faculty in seeing what each other is doing in teaching situations 
that are similar enough to their own content area to have some 
potential applicability. While generic principles of instruction 
are helpful, being knowledgeable in a content area is critical for 
generating content specific pedagogical knowledge and exploring the 
issues of what to teach and why. Even so, problems of the 
relevance of the feedback provided by mentors in portfolio review 
may persist. Faculty in the same general academic discipline 
frequently questioned whether the ideas for teaching content in 
other similar courses would be useful for their courses. 

Criteria to be used in evaluating portfolios was a major 
concern of faculty. Issues surrounding the development of criteria 
for evaluating portfolios are complex and unresolved. Edgerton, 
Hutchings and Quinlan (1991) suggest that criteria be developed for 
the critical tasks of teaching: planning and preparation, actual 
teaching, evaluating student learning and providing feedback and 
keeping up with the professional field in areas related to teaching 
performance. Seldin (1993b) suggests that teaching objectives be 
related to what is done in the course and that evidence of the 
effectiveness of these practices be referenced and documented in 
the teaching portfolio for the critical tasks of teaching just 
noted. He offers several suggestions for the kinds of information 
that might be provided to substantiate claims of effective teaching 
in each of these areas. Other approaches to developing criteria 
have focused on examining the institutional goals and the goals of 
the department in which the faculty member teaches and insuring 
that these are addressed in the faculty members portfolio. 
Port f ol ios are evaluated hoi ist ical ly , and the reults of that 
evaluation are reported as numerical ratings or as narrative 
comments . 

Most faculty members did not feel that they could make 
qualitative judgements on the dimensions of teaching noted above, 
and even those that felt they could did not feel comfortable doing 
so. They felt that differences were a matter of "style" or 
personal preference . Furthermore , some of the most important 
objectives of instruction, such as developing thinking skills, 
fostering multiple viewpoints in students , challenging their 
stereotypes, or enriching their appreciation of otfters are 
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difficult to readily change and we do not have adequate measures to 
show those changes. Some of these objectives are best justified as 
being of value in and of themselves, and difficulty in relating 
these to demonstrable changes in student learning may subvert 
attention to them in instruction. At this point in the development 
of portfolio review, evaluations of teaching portfolios for 
accounta? ility purposes is probably best left up to individuals who 
have to make those judgements such as chairs and those faculty 
members serving on promotion and tenure or awards committees. 

Differences between the mission and formal pedagogical 
training in the departments suggests that decisions about the 
purpose for using portfolios, what to include in them, and who 
might be involved in the process should be made at the 
departmental, programmatic, or even individual level. In 
departments such as accounting and dental hygiene where there is a 
common set of concerns, and a focus on professional preparation, 
and faculty are accepting of these common concerns, preparation of 
portfolio for purposes of accountability might be appropriate. 
Preparation of portfolios here could still be used to stimulate 
dialogue about effective teaching, capitalize on the benefit of 
putting these in writing, and help individual faculty members or 
the department as a whole make a case for effective teaching. 

Further considerations here would include whether the faculty 
members themselves could act as mentors to formulate effective 
teaching practices, or would find an outside consultant more 
beneficial. In the case of dental hygiene, the typical background 
training of faculty members in educational and adult learning 
theory would suggest they could profitably conduct this process on 
their own. Accounting, while incorporating many of the same 
processes and ideas in their teaching, might benefit from having 
help in translating or examining these ideas with knowledgeable 
outside consultants. 

Faculty in the sociology department have some common concerns 
of what to teach in general education courses but have diverse 
perspectives on what is important to teach in most of their 
courses. Also, they see the use of portfolios for accountability 
as restrictive and a threat to academic freedom. The rich and 
varied approaches to teaching used by faculty members of this 
department might best be represented through "episodes" of teaching 
that might evolve out of an evaluation process focusing on using 
portfolio review for improvement. This could be accomplished by 
having mentors from within or outside the department work with each 
other to construct episodes of teaching, and using ^eer review for 
providing feedback only to the faculty member. If issues of 
accountability later arise, faculty of that department could 
present a case for providing rich, varied and insightful 
experiences for students through examples of teaching episodes or 
a restructuring of their portfolios. Schackelford (1993) suggests 
that faculty may want to keep two kinds of portfolios, one for 
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"their eyes only" for improvement, and another that could be 
presented for others to evaluate for accountability. 

Non-tenured faculty may find the process of constructing 
portfolios very useful . Seldin (1993b) notes that tenure and 
promotion committees react favorably to reviewing well jstructured 
and substantiated cases for effective teaching. Providing 
reflective statements about teaching helps to contextual ize and 
focus the faculty members teaching accomplishments for review by 
tenure and promotion committees. 

SUMMARY AND RECOMMENDATIONS 

Whatever way departments choose to conduct portfolio review, 
there needs to be a recognition that it is a very time consuming 
process to construct, and especially to evaluate, portfolios. If 
faculty do not see the benefit of constructing portfolios or do not 
want to construct portfolios, it seems ill advised to force them to 
do so. The motivation for participating in portfolio review comes 
from participating in the process c.nd what faculty get from it. 
Faculty members can best evaluate that after they have attempted to 
use portfolio review. Administrative encouragement and support is 
essential to provide the motivation to try out portfolio review, 
and to provide the resources for some form of mentoring. If 
interest in teaching enhancement projects arise from that process, 
university level resources need to be available to respond to those 
interests. 

Implementing a system of portfolio review is difficult, and 
the practical suggestions available from Seldin and his associates 
(1991, 1993b); Edgerton, Hutchings, and Quinlan (1991); and 
Shulman and his associates working at Stanford are very important 
to successful implementation of a portfolio review process. Millis 
(1991) provides an excellent discussion and set of references for 
dealing with the kinds of issues faculty raise in constructing and 
reviewing teaching portfolios. I found the information gained from 
participating in the recent national symposium on portfolio review 
very helpful for interpreting what went wrong. The major benefit 
for improving instruction comes from the dialogue stimulated by 
portfolio construction and review, and from the requirement that 
processes and outcomes of effective teaching be captured in 
writing. Seldin (1993b) and Edgerton, Hutchings, and Quinlan 
(1991) note effective use of portfolios develops out of working 
with them in particular contexts and for particular purposes and 
there is no one formula for successful implementation. 

Seldin (1990) and othars (Edgerton, Hutchings & Quinlan, 1991) 
see the need for making a better case for effective teaching as a 
necessary response to the winds of change in higher education and 
the demand for better accountability. Teaching portfolios, while 
not perfect, can present a better case for effective teaching while 
preserving the individual, complex and transformational nature of 
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teaching, than any other existing form of evaluation. This would 
seem to be best accomplished by letting the process of portfolio 
review evolve out of the unique concerns of academic disciplines, 
leaving the form that portfolios might take flexible, and avoiding 
using them for accountability purposes too early on in the process. 
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